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Preface  &  Acknowledgements 


Welcome  to  our  Tenth  Annual  Acquisition  Research  Symposium!  We  regret  that  this 
year  it  wili  be  a  “paper  oniy”  event.  The  double  whammy  of  sequestration  and  a  continuing 
resoiution,  with  the  attendant  restrictions  on  travei  and  conferences,  created  too  much 
uncertainty  to  properly  stage  the  event.  We  will  miss  the  dialogue  with  our  acquisition 
coileagues  and  the  opportunity  for  ail  our  researchers  to  present  their  work.  However,  we 
intend  to  simuiate  the  symposium  as  best  we  can,  and  these  Proceedings  present  an 
opportunity  for  the  papers  to  be  pubiished  just  as  if  they  had  been  delivered.  In  any  case,  we 
will  have  a  rich  store  of  papers  to  draw  from  for  next  year’s  event  scheduied  for  May  14-15, 
2014! 


Despite  these  temporary  setbacks,  our  Acquisition  Research  Program  (ARP)  here  at 
the  Navai  Postgraduate  School  (NPS)  continues  at  a  normai  pace.  Since  the  ARP’s 
founding  in  2003,  over  1 ,200  originai  research  reports  have  been  added  to  the  acquisition 
body  of  knowiedge.  We  continue  to  add  to  that  library,  iocated  online  at 
www.acquisitionresearch.net,  at  a  rate  of  roughly  140  reports  per  year.  This  activity  has 
engaged  researchers  at  over  70  universities  and  other  institutions,  greatiy  enhancing  the 
diversity  of  thought  brought  to  bear  on  the  business  activities  of  the  DoD. 

We  generate  this  ievei  of  activity  in  three  ways.  First,  we  soiicit  research  topics  from 
academia  and  other  institutions  through  an  annual  Broad  Agency  Announcement, 
sponsored  by  the  USD(AT&L).  Second,  we  issue  an  annual  internal  call  for  proposals  to 
seek  NPS  faculty  research  supporting  the  interests  of  our  program  sponsors.  Finally,  we 
serve  as  a  “broker”  to  market  specific  research  topics  identified  by  our  sponsors  to  NPS 
graduate  students.  This  three-pronged  approach  provides  for  a  rich  and  broad  diversity  of 
scholarly  rigor  mixed  with  a  good  blend  of  practitioner  experience  in  the  field  of  acquisition. 
We  are  gratefui  to  those  of  you  who  have  contributed  to  our  research  program  in  the  past 
and  encourage  your  future  participation. 

Unfortunately,  what  will  be  missing  this  year  is  the  active  participation  and 
networking  that  has  been  the  hallmark  of  previous  symposia.  By  purposely  limiting 
attendance  to  350  people,  we  encourage  just  that.  This  forum  remains  unique  in  its  effort  to 
bring  scholars  and  practitioners  together  around  acquisition  research  that  is  both  relevant  in 
application  and  rigorous  in  method.  It  provides  the  opportunity  to  interact  with  many  top  DoD 
acquisition  officials  and  acquisition  researchers.  We  encourage  dialogue  both  in  the  formal 
panel  sessions  and  in  the  many  opportunities  we  make  available  at  meals,  breaks,  and  the 
day-ending  socials.  Many  of  our  researchers  use  these  occasions  to  establish  new  teaming 
arrangements  for  future  research  work.  Despite  the  fact  that  we  will  not  be  gathered 
together  to  reap  the  above-listed  benefits,  the  ARP  will  endeavor  to  stimulate  this  dialogue 
through  various  means  throughout  the  year  as  we  interact  with  our  researchers  and  DoD 
officials. 

Affordability  remains  a  major  focus  in  the  DoD  acquisition  world  and  will  no  doubt  get 
even  more  attention  as  the  sequestration  outcomes  unfold.  It  is  a  central  tenet  of  the  DoD’s 
Better  Buying  Power  initiatives,  which  continue  to  evolve  as  the  DoD  finds  which  of  them 
work  and  which  do  not.  This  suggests  that  research  with  a  focus  on  affordability  will  be  of 
great  interest  to  the  DoD  leadership  in  the  year  to  come.  Whether  you’re  a  practitioner  or 
scholar,  we  invite  you  to  participate  in  that  research. 

We  gratefully  acknowledge  the  ongoing  support  and  leadership  of  our  sponsors, 
whose  foresight  and  vision  have  assured  the  continuing  success  of  the  ARP: 
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•  Director,  Acquisition  Career  Management,  ASN  (RD&A) 

•  Program  Executive  Officer,  SHIPS 
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•  Program  Executive  Officer,  Integrated  Warfare  Systems 

•  Army  Contracting  Command,  U.S.  Army  Materiel  Command 
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•  Office  of  the  Assistant  Secretary  of  the  Army  (Acquisition,  Logistics,  & 
Technology) 
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•  Office  of  Procurement  and  Assistance  Management  Headquarters, 
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•  Deputy  Assistant  Secretary  of  the  Navy,  Research,  Deveiopment,  Test,  & 
Evaluation 

•  Program  Executive  Officer,  Tacticai  Aircraft 

•  Director,  Office  of  Small  Business  Programs,  Department  of  the  Navy 

•  Director,  Office  of  Acquisition  Resources  and  Anaiysis  (ARA) 

•  Deputy  Assistant  Secretary  of  the  Navy,  Acquisition  &  Procurement 

•  Director  of  Open  Architecture,  DASN  (RDT&E) 

•  Program  Executive  Officer,  Littorai  Combat  Ships 

James  B.  Greene  Jr.  Keith  F.  Snider,  PhD 

Rear  Admirai,  U.S.  Navy  (Ret.)  Associate  Professor 


ACQUISITION  RESEARCH  PROGRAM: 
CREATING  SYNERGY  FOR  INFORMED  CHANGE 


System  of  Systems  Management 


Acquisition  Management  for  System  of  Systems:  Affordabiiity  Through  Effective 
Portfolio  Management 

Navindran  Davendralingam  and  Daniel  DeLaurentis 
Purdue  University 

Identifying  Governance  Best  Practices  in  Systems-of-Systems  Acquisition 

David  J.  Berteau,  Guy  Ben-Ari,  Joshua  Archer,  and  Sneha  Raghavan 
Center  for  Strategic  and  Internationai  Studies 

The  Making  of  a  DoD  Acquisition  Lead  System  Integrator  (LSI) 

Paul  Montgomery,  Ron  Carlson,  and  John  Quartuccio 
Naval  Postgraduate  School 

Innovating  Naval  Business  Using  a  War  Game 

Nickolas  Guertin  and  Brian  Womble,  United  States  Navy 
Paul  Bruhns,  ManTech  International  Corporation 

Computer-Aided  Process  and  Tools  for  Mobile  Software  Acquisition 

Christopher  Bonine,  Man-Tak  Shing,  and  Thomas  W.  Otani 
Naval  Postgraduate  School 


ACQUISITION  RESEARCH  PROGRAM: 
CREATING  SYNERGY  FOR  INFORMED  CHANGE 


-624- 


The  Making  of  a  DoD  Acquisition  Lead  System  Integrator 

(LSI) 


Paul  Montgomery — After  retiring  in  1990  from  a  20-year  career  in  the  Navy,  Dr.  Paul  Montgomery 
served  as  a  senior  systems  engineer  with  Raytheon  and  Northrop  Grumman  corporations  and 
developed  communications,  surveillance,  and  sensor  systems  for  commercial,  military  (USN,  USA, 
USAF),  and  intelligence  communities  (NSA,  NRO).  He  earned  his  doctorate  in  systems  engineering 
from  George  Washington  University  (D.Sc.  2007)  performing  research  related  to  cognitive/adaptive 
sensors,  his  MSEE  (1987)  from  the  Naval  Postgraduate  School,  and  his  BSEE  (1978)  from  Auburn 
University.  The  International  Council  on  System  Engineering  (INCOSE)  certifies  him  as  an  Expert 
Systems  Engineering  Professional  (ESEP).  Montgomery  is  an  embedded  faculty  member  in  the  SE 
department  providing  onsite  research  and  instruction  support  to  NAVAIR  (Patuxent  River,  MD), 
NAVSEA  (Dahlgren,  VA,  Carderock,  MD),  and  NPS  SE  students  in  the  Nation  Capital  Region. 
[prmontgo@nps.edu] 

Ron  Carlson — Carlson  served  26  years  in  naval  aviation  as  a  pilot,  seven  years  of  which  were  at 
NAVAIR  where  he  led  NAVAIR  systems  engineers  through  several  years  of  systems  engineering 
revitalization  to  the  NPS  SE  department.  He  is  currently  in  the  systems  engineering  doctoral  program 
at  Stevens  Institute  of  Technology.  He  earned  master’s  degrees  in  strategic  studies  and  national 
policy  from  the  Naval  War  College,  a  master’s  degree  in  business  administration-aviation  from  Embry 
Riddle  Aeronautical  University,  and  his  Bachelor  of  Science  in  nuclear  engineering  from  the 
University  of  Michigan.  Carlson  is  an  embedded  faculty  member  in  the  SE  department  providing 
onsite  research  and  instruction  support  to  NAVAIR  (Patuxent  River,  MD),  NAVSEA  (Dahlgren,  VA, 
Carderock,  MD),  and  NPS  SE  students  in  the  Nation  Capital  Region,  [rrcarlso@nps.edu] 

John  Quartuccio — Quartuccio  has  more  than  29  years  of  civilian  service  within  the  Naval  Air 
Systems  Command  and  the  Naval  Air  Warfare  Center.  He  is  a  graduate  from  The  Pennsylvania  State 
University,  with  a  Bachelor  of  Science  in  Mechanical  Engineering  in  1985,  and  a  graduate  of  Lehigh 
University,  with  a  Master  of  Science  in  Applied  Mechanics  in  1997.  He  is  currently  an  NPS  systems 
engineering  PhD  student.  Quartuccio  is  the  director  of  the  Systems  Engineering  Development  and 
Implementation  Center  (SEDIC)  within  Air  Platform  Engineering  (AIR-4.1 .1)  of  the  Systems 
Engineering  Department  and  a  member  of  the  AIR-1 .0  staff  as  APEO(E)  for  AIR-1 .0  Programs. 
Ojquartu@nps.edu] 

Abstract 

The  complexity  of  developing  and  acquiring  weapons  systems  continues  to  increase  due  to 
highly  integrated  system  architectures,  rapid  technology  evolution,  and  emergence  of  highly 
diverse  set  of  missions.  The  imperatives  of  system-of-systems  (SoS)  integration  and 
interoperability  (l&l)  further  complicate  the  system  acquisition  process.  These  challenges 
continue  to  frustrate  completing  the  acquisition  of  systems  within  time  and  budget  goals. 

The  DoD  has  commonly  assigned  the  role  of  the  lead  system  Integrator  (LSI)  to  a  prime 
contractor.  This  is  fraught  with  many  issues  related  to  conflict  of  interest,  performance,  and 
defining  clear  roles  and  responsibilities  (especially  the  inherent  role  of  government).  The  DoD 
has  indicated  that,  in  some  cases,  the  LSI  responsibilities  should  migrate  back  to  the  DoD. 

In  this  paper,  we  discuss  the  roles  of  the  LSI,  where  DoD  acquisition  skills  may  need  to  be 
strengthened  to  perform  as  the  LSI,  and  discuss  methods  and  tools  to  do  so.  This  paper  is  a 
result  of  multi-year  discussions  and  research  with  a  major  Naval  Systems  Command  to  find  a 
path  to  faster  time-to-market  and  higher  levels  of  interoperability  and  integration  of  our 
weapons  system  acquisitions. 
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Introduction 

What  is  a  lead  system  integrator  (LSI)?  Although  a  broader  concept  than  its  simple 
acronym,  commonly,  the  role  of  the  LSI  has  been  turned  over  to  industry  in  the  form  of  a 
prime  contractor,  or  team  of  contractors,  hired  by  the  federal  government  to  execute  a  large, 
complex,  system-of-systems  (SoS),  defense-related  acquisition  programs  (Grasso,  2007). 
The  need  for  an  LSI  is  often  associated  with  the  acquisition  of  an  SoS  or  a  constituent 
system  to  an  SoS.  SoS  programs  acquire  a  collection  of  various  platforms  (e.g.,  ground 
vehicles,  aircraft,  and  ships)  that  are  to  be  linked  together  so  as  to  create  a  larger, 
integrated  overall  system  (Lane,  2006). 

LSIs  are  further  categorized  based  on  their  system  development  capabilities  and 
responsibilities.  Section  805  of  the  National  Defense  Authorization  Act  of  Fiscal  Year  2006 
(2005)  defines  these  two  types  of  LSIs  as 

1 .  “Lead  system  integrators  with  system  responsibility”  prime  contractors  who 
develop  major  systems  that  are  not  expected  at  the  time  of  the  contract 
award,  as  determined  by  the  Secretary  of  Defense,  to  perform  a  substantial 
portion  of  the  work  on  the  system  and  major  subsystems. 

2.  “Lead  system  integrators  without  system  responsibility” — contractors  who 
perform  acquisition  functions  that  are  closely  associated  with  inherently 
governmental  functions  in  the  development  of  a  major  system.  LSIs, 
regardless  of  type,  are  subject  to  the  same  rules  as  other  federal  contractors. 

In  recent  years,  the  LSI  responsibility  has  been  awarded  to  industry  for  major  DoD 
acquisitions.  However,  this  has  led  to  conflict-of-interest  complications  resulting  in  revised 
law  stating,  “No  entity  performing  lead  systems  integrator  functions  in  the  acquisition  of  a 
major  system  by  DoD  may  have  any  direct  financial  interest  in  the  development  or 
construction  of  any  individual  system  or  element  of  any  system  of  systems”  (Defense 
Authorization  Act  for  Fiscal  Year  2006,  Section  807).  As  a  result,  several  of  the  major 
contractors  have  divested  into  companies  focused  separately  on  systems  integration  and 
product  development.  An  example  is  Lockheed  Martin,  where  “Lockheed  Martin’s  decision 
to  divest  the  business  was  based  on  the  U.S.  Government's  increased  concerns  regarding 
perceived  conflicts  of  interest”  (The  SI  Organization,  2010). 

Due  to  recent  failures  in  some  major  DoD  acquisition  programs  (examples  in  GAO, 
2007),  the  DoD  has  made  the  decision  to  use  an  LSI  endure  more  scrutiny  by,  in  some 
cases,  requiring  certification  by  the  Committees  on  Armed  Services  for  both  the  Senate  and 
the  House  (OSD,  2007).  This  has  led  some  to  conclude  that  to  reduce  complexities  and 
risks  associated  with  the  use  of  contractors  as  the  LSIs,  the  DoD  should  consider  (Grasso, 
2007): 

•  prohibiting  the  use  of  private-sector  LSIs  in  future  acquisition  programs; 

•  reducing  the  possible  need  for  private-sector  LSIs  by  building  the  defense 
civilian  and  military  acquisition  workforces  back  up,  and  having  the  DoD 
assume  the  role  of  the  LSI,  and  requiring  that  DoD  manage  all  SOS 
programs;  and 

•  implementing  the  recommendations  of  the  Gansler  Commission  on  improving 
the  acquisition  workforce  (U.S.  Army,  2007). 

The  following  discussion  begins  with  the  premise  that  the  DoD  concurs  with  the 
recommendations  above  and  desires  to  bring  more  LSI  responsibilities  “in  house,”  in 
particular,  engineering  responsibilities.  Some  of  the  major  systems  commands  are  exploring 
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such  a  transformation  to  bring  systems  to  the  DoD  more  quickly  while  attaining  higher  levels 
of  interoperability  (Young,  2010).  If  the  DoD  acquisition  community  wants  to  make  such  a 
transformation  to  retain  more  inherently  governmental  responsibilities  for  major  system 
acquisitions,  what  needs  to  be  done  to  fortify  the  systems  engineering  (SE)  workforce  skills, 
SE  methods,  and  SE  tools  to  enable  taking  on  the  larger  role  of  the  LSI?  We  use  Figure  1 
as  a  context  diagram  throughout  (blue  is  our  focus). 


I 


DoD  Acquisition 
Cycle 


LSI  Management 
Roles 

T- 


-  LSI  SE  Roles 


L 


•-  SE  Processes 

- 

SOEA  Methods 
and  Tools 


Figure  1.  LSI  Systems  Engineering  and  Management  Roles  Are  Supported 
by  Systems  Engineering  Processes,  Methods,  and  Tools 

In  our  previous  research  (Montgomery,  Carlson,  &  Quartuccio,  2012),  we  focused  on 
how  SE  tools  could  be  applied  to  DoD  acquisition  SE  methods.  We  introduced  a  model- 
base,  SE-inspired  approach  named  System  Definition-Enabled  Acquisition  (SDEA)  in  that 
research  and  discussed  how  SDEA  could  be  instrumental  for  LSI  SE  success.  What  follows 
extends  that  previous  SE  tools  perspective  to  the  role  of  the  LSI  SE  as  a  result  of  ongoing 
research  with  Naval  Air  Systems  Command  (NAVAIR). 


Problem  Definition  and  Research  Questions 

The  Defense  Authorization  Act  for  Fiscal  Year  2006  (Section  807)  provides  emphasis 
on  the  importance  that  the  lead  systems  engineer  on  a  DoD  acquisition  be  an  experienced 
government  employee.  Additionally,  the  DoD  ASD(R&E)  chief  systems  engineer,  Stephen 
Welby  (2012)  summarized  the  imperatives  for  DoD  SE  as  follows: 

As  the  complexity  of  our  systems  has  increased,  so  has  the  need  for  effective 
systems  engineering  throughout  the  life  cycle.  We  face  challenges  in 
implementing  robust  systems  engineering  processes,  from  requirements 
identification  and  analysis  through  technology  and  architecture  selection  and 
assessment,  analysis  and  coordination  of  complex  system  design, 
development,  and  execution  ....  We  are  now  increasingly  focused  on 
addressing  early-acquisition  phases  Including  requirements  definition, 
development  planning,  and  early  acquisition  systems  engineering  support. 

Finally,  as  stated  in  a  report  sponsored  by  Welby  (Systems  Engineer  Research 
Center  [SERC],  2010),  “existing  systems  engineering  tools,  processes,  and  technologies 
poorly  support  rapid  design  changes  or  capability  enhancements  within  acceptable  cost  and 
schedule  constraints.” 


Problem  Statement 

The  background  and  guidance  presented  in  the  previous  section  leads  to  our 
investigation,  focused  by  the  following  problem  statement:  The  DoD  does  not  have 
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adequate  SE  methods,  processes,  workflows,  and/or  tools  that  support  the  expansive  role  of 
the  LSI  in  major  weapons  systems  acquisitions. 

Research  Questions 

The  associated  research  questions  that  we  have  been  investigating  in  order  to 
resolve  our  problem  statement  include  the  following: 


•  What  are  key  DoD  acquisition  challenges  for  the  LSI? 

•  What  are  the  key  LSI  roles  and  attributes? 

•  What  is  the  current  state  of  DoD  LSI  maturity? 

•  What  SE  methods  are  prime  candidates  to  improve  upon  to  support  LSI? 

•  How  can  MBSE/SDEA  be  applied  to  the  LSI? 

LSI  Challenges 

Regardless  of  the  government/contractor  ownership  of  LSI  SE  responsibilities,  the 
challenges  to  current  acquisitions  are  diverse,  not  necessarily  new,  and  are  discussed  as 
follows  (derived  from  Montgomery  et  al.,  2012). 


Complex  System  Acquisition 

The  current  DoD  acquisition  process  (see  Figure  2),  as  specified  in  DoD  5000, 
WSARA,  and  a  long  heritage  of  acquisition  experience,  is  based  on  the  acquisition  of  stand¬ 
alone  systems.  Today’s  system  acquisitions  are  more  co-dependent  on  the  development  of 
other  complex  systems  in  an  SoS  environment.  This  requires  a  higher  level  of  coupling 
between  system  engineering  and  the  acquisition  process  to  support  SoS,  as  well  as  the 
need  for  higher  levels  of  LSI  support. 


'  -  S-r  r. :  -  - 

SDEA 

'lid  TO--‘ 


Concept  Development  & 

Validatton 

Design  Development  & 

Valldetion 

Produce  &Qualify 

Deploy  &  Improve 


Figure  2.  DoD  5000  Acquisition  Process 

A  problem  that  continues  to  frustrate  this  acquisition  timeline  and  increase  program 
costs  is  both  system  complexity  and  SoS  interoperability.  Many  acquisitions  are  the 
integration  of  several  systems  that  are  being  acquired  and  developed  independently  and  for 
their  own  purposes.  This  SoS  method  presents  a  new  and  interesting  level  of  complexity  for 
system  engineers  because  system  engineers  rarely  have  the  opportunity  to  affect  the  design 
of  these  co-dependent  systems.  The  functionality,  interfaces,  operational  objectives,  and 
intended  system  environments  all  provide  a  challenge  to  ensuring  that  the  SoS  can  be 
integrated  successfully  while  producing  new  emergent  behaviors  that  are  predictable  and 
satisfy  the  user  needs.  Couple  all  of  this  complexity  and  SoS  realities  to  the  existing  system 
engineering  methods,  practices,  principles,  organization  behaviors,  and  workforce  skills,  and 
the  need  for  SDEA  methods  and  tools  becomes  clearer  to  resolve  many  of  the  following 
challenges. 
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Acquisition  Timeiiness 

Acquisitions  are  too  siow-to-market.  Acquisition  schedules  are  often  document- 
driven  and  technical  review-driven  processes  and  non-adaptive  to  changing  or  emergent 
requirements.  DoD  5000  emphasizes  prototypes  early  in  acquisition,  requiring  a  tightly 
coupled  engineering  system  to  meet  engineering  goals,  objectives,  and  requirements.  The 
LSI  SE  needs  to  be  diligent  to  ensure  pre-planned  programmed  improvements  (P3ls)  are 
enabled  and  that  tools  provide  enduring  design  data. 

Acquisition  Process 

The  Acquisition  process  is  not  LSI  design-driven.  The  DoD  5000  acquisition  process 
is  oversight-driven  and  document-driven  and  designed  such  that  government  engineers 
provide  the  oversight  while  the  contractors  provide  the  content.  It  is  likely  that  DoD  5000  will 
ultimately  have  to  be  revised  to  define  a  process  more  aligned  to  the  government  providing 
LSI  SE  direction.  An  example  would  be  to  exploit  a  process  that  that  could  be  streamlined 
as  a  result  of  the  government  and  user  community  retaining  more  LSI  SE  activities  and 
direct  engagement  with  the  development  of  the  baseline  in  lieu  of  merely  reviewing  the 
progress  of  the  contract. 

System  Complexity 

LSI  engineering  capabilities  do  not  always  support  design  and  acquisition  of  highly 
complex  systems.  Simple  systems  and  complex  systems  proceed  through  the  acquisition 
process  essentially  the  same.  The  role  of  the  LSI,  however,  is  more  applicable  for  the  needs 
of  complex  systems  with  a  significant  emphasis  on  defining  the  interaction  of  systems  along 
with  robustness  of  the  system  solution.  This  will  require  a  dramatically  different  way  of 
defining  the  LSI  engineering  process  and  how  it  integrates  with  the  program  management 
processes.  An  example  is  employment  of  tools  and  methods  to  provide  the  ability  to  assess 
SoS  performance  and  emergent  system  behaviors  in  a  quantifiable  manner.  Currently,  the 
ability  to  predict,  manage,  and  control  such  emergent  behaviors  can  be  elusive. 

Integration  and  Interoperability 

Systems  often  fail  at  integration  or  do  not  interoperate  effectively.  Successful 
integration  of  systems,  especially  SoS,  is  challenged  by  functional  gaps  and  overlaps 
among  the  systems’  complex  interfaces  and  a  large  number  of  internal  and  external  system 
interfaces.  SoS  integration  also  demands  the  interoperability  among  these  systems,  as  well 
as  the  interoperability  outside  of  the  system  for  other  systems  that  are  codependent.  The 
LSI  needs  SE  tools  and  methods  that  define  and  manage  risks  associated  with  these  critical 
functions  and  interfaces. 

Total  Ownership  Costs 

Prediction  and  control  of  total  ownership  costs  (TOC)  is  difficult.  The  acquisition  cost 
incurred  during  the  development  cycle  is  only  a  fraction  of  the  total  ownership  cost  of  any 
system.  The  LSI  needs  to  have  very  detailed,  predictable,  and  repeatable  behavior  modeling 
of  both  the  acquired  system  and  external  systems  in  order  to  accurately  predict  and  control 
TOC. 

Engineering  Workforce 

The  veteran  engineers  are  rapidly  retiring  and  not  being  replaced  with  engineers  with 
commensurate  experience.  The  system  design  process  and  SE  tools  need  to  provide  high 
levels  of  repeatability  and  quantifiability  that  is  less  dependent  on  engineering  judgment  and 
more  dependent  on  metrics  that  provide  a  highly  refined  engineering  solution.  Given  that 
many  veteran  engineers  are  retiring,  there  is  a  need  to  provide  system  design-driven 
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methods  to  a  younger  engineering  community.  A  system  is  needed  that  also  provides 
project-to-project  consistency  and  repeatability. 

Systems  Engineering  Attributes  and  Roles  of  an  LSI 

LSI  Attributes 

Not  all  DoD  acquisitions  need  to  be  managed  by  an  LSI.  Many  systems  can  be 
acquired  with  small  teams  where  complexity  and  risk  are  relatively  easy  to  manage.  The 
following  is  a  list  that  includes  attributes  of  program  and  system  designs  where  the  need  for 
an  LSI  may  be  more  imperative  (partially  derived  from  Loudin,  2010): 

•  Program  importance  and  span  of  impact — high  risk,  large  cost,  or  expansive 
interoperability  impacts  to  the  enterprise 

•  System  or  SoS  complexity — large-scale  complexity  with  a  large  number  of 
high-consequence  risks,  external  SoS  interfaces  and  interactions,  and  high 
likelihood  of  unanticipated,  negative  emergent  behaviors 

•  Stakeholder  relationships — Collaborative  versus  command-and-control 
contractor/government/fleet  user  interactions  are  necessary 

•  Organization  agility  is  required  to  organize  around  acquisition  (versus  the 
obverse) 

•  DoD  determines  that  ownership  of  critical  data  and/or  DoD  reuse  of  critical  IP 
is  mandated 

•  “National  teaming”  is  required  to  ensure  enterprise-level  SoS  issues  are 
intrinsic  to  system  success 

•  Acknowledgement  and  acceptance  of  higher  system  design  and  acquisition 
risks 

•  Rapid  identification  and  adaptation  of  emergent  opportunities  are  essential 

•  Strong  integration  leadership  and  control  is  required 

•  Low  barriers  to  entry  for  technology  and  innovation  need  to  be  established 
and  maintained  throughout  the  life  cycle 

•  Disciplined  and  rigorous  standards  demanded  for  integration  of  other  systems 
into  the  enterprise 

LSI  Roles 

The  roles  of  the  LSI  are  similar  to  the  roles  of  any  SE  or  system  integrator  (SI).  The 
primary  difference  is  the  span  of  design  and  integration  authority  that  persists  throughout  the 
system  acquisition  and/or  complete  life  cycle.  The  following  are  a  sampling  of  the  LSI  roles 
that  are  more  expansive  than  traditional  SE/SI: 

•  Design:  Act  as  the  primary  designer  (sometime  referred  to  as  the  “design 
agent”).  Design  includes  system  and  SoS  designs.  Roles  include  conceptual 
design,  architectural  design  (operational,  functional,  physical,  interface, 
qualification),  and  integration  and  qualification  designs. 

•  Source  selection:  Responsible  for  providing  solicitation  packages,  reviewing 
and  evaluating  proposals,  and  selecting  and  awarding  the  contract  to 
component,  subsystem,  system,  or  product  provider.  Component-level 
solicitation  has  often  been  assigned  to  prime  contractors. 
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•  Subcontractor  selection:  Survey,  vetting,  and  selection  of  providers  of 
components  or  services.  Component-level  selection  has  often  been  assigned 
to  prime  contractors. 

•  Supplier  chain  management:  Engagement  within  the  domains  of  hardware 
and  software  configuration  item  selection,  sources  of  supply,  and 
manufacture. 

•  T rade-off  studies:  Conduct  of  objective  trade-off  studies  and  analysis  of 
system  challenges,  risks,  and  opportunities. 

•  System  baseline  management:  Definition,  control,  and  management  of 
system  design  baselines,  configuration  management,  and  realized 
configurations. 

•  Rigorous,  multi-system  definition  and  management  of  interfaces,  taxonomy, 
system  structures,  and  so  forth. 

•  Coordinator  (and  funder)  of  contributing  research. 

•  End-to-end  span  of  authority  and  control  for  baseline  control  and 
management  of  the  system  design,  development,  integration,  qualification, 
and  deployment. 

•  Qualification  (“V&V”):  Ultimate  responsibility  for  developmental  (verification), 
operational  (validation),  and  acceptance  qualification  success. 

•  Sustainment/suitability:  Responsible  for  sustainment  and  suitability  design  of 
the  system  and  impact  analysis  of  SoS  sustainment  strategies. 

Current  State  of  the  DoD  LSI 

How  do  many  engineering  organizations  operate  today  in  DoD  acquisition?  There 
have  been  many  strains  on  DoD  manpower  reductions  over  recent  years,  and  the  result  has 
been  to  depend  heavily  on  contractors  to  do  the  “heavy  lifting”  in  many  engineering 
domains.  Although  the  government  retains  many  subject-matter  experts  (SMEs),  these 
highly  skilled  staff  are  senior,  retiring  at  a  rapid  rate,  and  are  stretched  thin.  The  larger  and 
more  complex  the  project,  the  more  likely  the  government  has  decided  to  use  a  large  prime 
or  LSI  contractor. 

The  different  roles  of  engineering  involvement  are  shown  in  Figure  3,  spanning  from 
performing  the  role  of  the  “buyer”  for  simple  systems  to  the  role  of  “integrator”  when 
acquiring  complex  systems.  (We  put  forth  these  role  titles  just  to  provide  reference;  they  are 
not  formally  accepted  throughout  DoD).  As  can  be  seen,  the  engineering  tasks 
(requirements  engineering,  design,  etc.)  become  more  expansive  as  the  role  approaches 
that  of  the  LSI  (integrator).  The  roles  close  to  red  (bottom  of  the  list)  are  associated  with 
complex  acquisitions.  Our  assumption  is  that  the  government  has  been  performing  in  the 
yellow  band  of  this  chart  during  recent  years.  As  previously  stated,  contractors  have  been 
more  likely  to  be  assigned  the  majority  of  engineering  duties  as  the  systems  became  more 
complex. 
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Figure  3.  The  Engineering  Involvement  of  the  DoD  Acquirer  Becomes  More  Expansive 
for  Complex  (“Integrator”  Role)  Systems  as  Compared  to  Simple  (“Buyer” 

Role)  Systems 

Figure  4  depicts  “traditional”  versus  “LSI”  contractor/government  engineering  span  of 
authority  across  the  DoD  acquisition  cycle.  The  top  portion  (a)  is  a  typical  acquisition  cycle 
that  spans  from  system  concept  to  deployment.  The  middle  portion  (b)  indicates  that  the 
traditional  government  levels  of  engineering  effort  are  maximum  at  the  early  and  latter 
stages  of  the  acquisition,  with  the  contractor  design,  production,  and  integration  in  the 
middle.  The  lower  portion  (c)  of  the  diagram  posits  that,  if  the  government  is  the  LSI,  the 
government  performs  more  of  the  design  and  integration  activities,  and  the  contractor  shifts 
to  a  more  “manufacturing”  role.  Although  there  could  be  many  variations  on  this  LSI  theme, 
what  is  important  to  note  is  that  the  area  under  the  curve  represents  the  government  level  of 
effort.  Some  refer  to  his  role  as  the  “design  agent.”  In  the  LSI  case,  this  level  of  effort  is 
more  expansive  than  today  and  requires  new  methods,  practices,  and  tools  to  support  the 
government  engineer. 
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Figure  4.  The  LSI  Roles  for  the  Government — (c)  Compared  to  (b) — Will  Require  Greater 
Methods,  Practices,  and  Toois  to  Achieve  Work  of  the  Increased  “Area  Under 

the  Curve” 

Another  perspective  is  to  try  to  assess  where  the  “maturity”  of  the  government 
engineering  community  (writ  large)  is  today  and  how  it  needs  to  be  enhanced.  Figure  5  puts 
forward  a  non-scientific  assessment  of  where  that  maturity  may  be  (dotted  line).  The  colors 
align  with  Figure  3  and  shows  that  the  current  DoD  acquisition  workforce  performs 
comfortably  as  a  “buyer”  and  “purchaser”,  often  at  the  “acquirer”  level,  but  has  yet  regularly 
perform  at  the  “architect/designer”  or  “integrator/LSI”  role.  The  graph  shows  that,  as  the 
government  makes  the  transition  to  the  upper  right  of  the  graph,  the  engineering  and 
programmatic  span  of  authority  must,  necessarily,  increase  for  the  government  and 
decrease  for  the  contracting  community. 
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DoD  acquisition  engineering 
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Figure  5.  Increasing  Role  of  Government  Engineering  Toward  “Integrator”  Will  Involve 
Reducing  Span  of  Design  and  Programmatic  Authority  for  the  Contractor 


Systems  Engineering  Methods  Supporting  LSI 

There  are  probably  very  few  new  fundamental  principles  and  essential  activities  that 
are  required  for  the  LSI;  however,  the  depth  and  ownership  of  SE  activities  are  greater  and 
more  enduring.  In  order  for  the  DoD  to  move  to  the  upper  right-hand  corner  of  Figure  5, 
additional  SE  practices,  methods,  and  tools  need  to  be  enhanced.  A  representative  SE 
activity  set  typically  employed  throughout  any  system  acquisition  cycle  is  shown  in  Figure  6. 
Although  we  can  anticipate  that  many,  if  not  all,  of  the  activities  will  be  impacted  by  taking  on 
the  role  of  the  LSI,  our  interviews  have  indicated  that  the  early  application  of  discipline  SE 
practices  and  methods  create  the  greatest  and  most  significant  positive  impact  to  reducing 
risk  and  increasing  system  success.  Figure  6  shows  that  the  dark  blue  activities  are  the 
most  likely  candidates  to  receive  attention  for  workforce  development  and  to  apply  SDEA 
concepts  and  tools. 
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Figure  6.  Systems  Engineering  Activities  Need  Strengthening  to  Expand  Role  as  an  LSI 


Although  still  formative,  the  SE  activities  shown  may  focus  upon  general  concepts  of 
the  following: 

•  Concept  development — Eliminating  disconnects  between  originator  needs 
and  acquisition  system  requirements. 

•  Design — Inexperience,  insufficient  or  missing  tools,  and  weak  methods. 

•  Integrative  methods — Organizational  teaming,  SoS  awareness,  standards, 
priorities,  technical  incentives. 

•  Development — DoD  (LSI)  and  contractor  common  models  and  tools. 

•  Integration — Gaps  in  cross-discipline  skills  and  experience,  lack  of  facilities, 
weak  methods,  lack  of  jointness. 

•  Test  and  evaluation — Gaps  in  attaining  a  system  that  is  mature  and  ready  for 
test. 

System  Definition-Enabled  Acquisition  (SDEA)  System  Concept 
Top-Level  Concept 

The  top-level  SDEA  concept  is  shown  in  Figure  7.  The  intent  is  that  SDEA  supports 
all  of  the  SE  activities  in  Figure  6  in  a  quantitative  and  repeatable  manner.  The  SDEA 
system  comprises  system  definition,  modeling,  and  analyses  that  provide  repeatable  and 
quantifiable  designs.  The  SDEA  system  is  to  provide  a  data-driven  system  definition  and 
model-driven  SE  approach  that  supports  LSI  SE  and  design. 

The  SDEA  system  is  synergistic  with  the  program  definition,  system  definition, 
supportability  definition,  and  system  production.  Note  that  all  of  these  activities  support 
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baseline  development  and  control.  Program  definition  leads  to  system  definition  and  the 
handoff  contract  (documents)  associated  with  system  capabilities  and  top-level  performance 
goals. 


Additionally,  program  definition  leads  to  documentation  and  agreements  that  set  in 
motion  long-term  supportability  strategies  and  activities,  such  as  logistics,  training  and 
manpower,  and  long-term  supply  chain  strategies.  The  SDEA  system  supports  both  system 
definition  in  a  very  repeatable  and  quantifiable  manner,  as  well  as  providing  clear  detail  and 
system  reliability  and  supportability  metrics  to  the  support  system  associated  with  the 
acquisition. 


Finally,  system  production  depends  on  precise  SDEA  system  definition  in  order  to 
proceed  to  production  of  the  system  in  preparation  for  deployment. 
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Figure  7.  SDEA  Provides  Central  Engineering  System  Support  to  Acquisition 

(derived  from  Montogmery  et  al.,  2012) 


Summary 

Past  performance  by  contractors  performing  what  some  now  believe  are  inherently 
governmental  acquisition  engineering  functions  during  major  DoD  weapon  system 
acquisition  has  proven  problematic,  in  some  cases.  Legislation  and  policy  is  moving  DoD  to 
consider  transforming  its  engineering  role  for  major  systems  (especially  SoS)  to  that  of  the 
LSI.  The  DoD  acquisition  workforce  methods,  practices,  and  tools,  however,  need  to  be 
upgraded  and  enriched  to  achieve  this  transformation.  We  believe  that  the  integration  of 
model-based  system  engineering  (MBSE)  tools  through  an  SDEA  method  is  key  to 
supporting  the  higher  levels  of  SE  design  disciplines,  analyses,  and  baseline  control,  and 
will  contribute  to  quicker  time-to-market  and  lower  integration  and  interoperability  risks  for 
future  weapons  systems. 


In  summary: 


•  An  LSI  Is  needed  where  high  system  complexity,  high  risks,  or  SoS 
integration/interoperability  are  present. 

•  DoD  acquisition  organizations  are  exploring  taking  on  more  of  the  LSI  roles. 

•  DoD  acquisition  practices  need  fortifying  to  cope  with  the  more  expansive 
levels  of  SE. 
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•  SE  methods  need  reinforcing,  and  SE  tools  (e.g.,  SDEA)  need  to  be  acquired 
and  integrated  into  the  workflow  with  capability  to  provide 

o  early  and  strong  SE  application  (pre-milestone  A), 

o  data-driven  design  support  tools, 

o  repeatable  and  quantifiable  system  design  analyses, 

o  persistent  (multi-year/multi-system)  design  data  repository, 

o  SoS  interoperability  and  integration  analyses,  and 

o  operational,  qualification  (V&V),  suitability,  and  sustainability 
design/analysis  support. 
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